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A  MODEL  FOR  INCORPORATING  RESPONSE-TIME  DATA  IN  SCORING 

ACHIEVEMENT  TESTS 


Kikumi  Tatsuoka  and  Maurice  Tatsuoka 
University  of  Illinois 


ABSTRACT 


The  differences  in  type  of  information-processing  skill  developed 
by  different  instructional  backgrounds  affect,  negatively  or  positively, 
the  learning  of  further  advanced  Instructional  materials.  That  is,  if 
prior  and  subsequent  instructional  methods  are  different,  a  proactive 
inhibition  effect  produces  low  achievement  scores  on  a  posttest.  This  fact 
poses  a  serious  problem  for  routing  of  students  to  an  instructional  level 
on  the  sole  basis  of  performance  on  a  diagnostic  adaptive  test.  It  is 
essential  that  we  somehow  unravel  what  information-processing  strategy  was 
used  and  consider  this  knowledge  simultaneously. 

Fortunately,  response  time  often  provides  supplementary  information 
which  differentiates  among  individuals  showing  identical  quality  of 
performance.  A  model  that  reflects  this  kind  of  information,  obtainable 
from  response  time  scores,  is  formulated  in  a  similar  manner  to  latent 
trait  theory  and  is  discussed.  This  model  is  useful  in  identifying 
discriminating  items  that  are  sensitive  to  differences  in  instructional 
method.  It  also  is  helpful  in  identifying  an  individual's  instructional 
background  to  a  certain  extent. 
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INTRODUCTION 

The  study  by  Tatsuoka  and  Birenbaum  (1979)  raised  an  important 
issue  with  respect  to  adaptive  diagnostic  testing  and  computer-managed 
routing  by  which  each  examinee  is  sent  to  his/her  level  of  instruction.  It 
is  necessary  to  consider  an  alternative  scoring  procedure  in  which 
individual  differences  in  information-processing  skills  have  been  taken 
into  account  along  with  individual  ability  or  achievement  levels. 

In  Tatsuoka  and  Birenbaum's  study,  a  computerized  diagnostic 
adaptive  test  for  a  series  of  pre-algebra  signed-number  lessons  was  given 
to  eighth-graders  at  a  junior  high  school,  and  a  computer-managed  routing 
system  sent  each  examinee  to  the  instructional  unit  corresponding  to  the 
level  of  skill  at  which  he/she  ended  up  in  the  initial  test.  The  adaptive 
test  for  signed-numbers  consisted  of  12  groups  of  items  representing  12 
different  skills.  The  instructional  units  of  computerized  lessons  that 
teach  the  same  12  skills  were  rearranged  into  the  same  order  as  the  skills 
in  the  adaptive  test,  so  that  if  an  examinee  stopped  at  the  seventh  skill 
level,  he/she  was  sent  to  the  seventh  level  of  the  lessons.  After  this 
student  went  through  the  7th  to  12th  instructional  units,  a  52-item, 
conventional  computerized  posttest  was  administered. 

Factor  analysis  revealed  that  the  test  scores  of  the  posttest  were 
far  from  satisfying  the  assumption  of  local  independence — i.e., 
unidimensionality.  A  further,  close  investigation  was  done  by  performing  a 
cluster  analysis  on  the  92  examinees'  response  patterns  on  the  basis  of 
Euclidean  distances  between  pairs  of  response  vectors.  The  result  of  this 
analysis  led  us  to  find  a  group  of  students  whose  response  patterns  were 


PHECKD1NG  FAOB  bLAHK 


4 


significantly  different  from  others.  Their  scores  on  the  items  prior  to  the 
stopping  level  of  the  initial  diagnostic  test  were  higher  than  most  scores 
of  other  students,  while  their  scores  on  the  the  subsequent  items  were 
as  low  as  the  poorest  students'  scores.  We  confirmed  with  their  teachers 
that  most  of  then  were  actually  A-students.  We  also  confirmed  that  the 
members  of  this  group  were  taught  signed-number  addition  operations  by  a 
teaching  method  different  from  that  of  subsequent  instructional  units  which 
teach  subtraction  operations.  The  procedure  of  information  processing 
associated  with  these  two  instructional  methods  of  performing  arithmetic 
upon  signed  numbers  are  greatly  different.  The  traditional  scoring 
procedure  of  the  latent  trait  theory  would  not  be  capable  of  detecting 
these  disc repancies  associated  with  different  information  processes  for 
arriving  at  the  answers  to  a  given  item. 

A  study  by  Tatsuoka  and  Tatsuoka  (1978)  Indicates  one  useful 
approach  toward  the  goal  just  mentioned.  They  showed  that  under  certain 
general  conditions,  item  response  time  scores  very  closely  follow  Weibull 
distributions — a  three  parameter  family  extensively  used  in  system 
reliability  theory  (see,  e.g.,  Mann  et  al«.  1974).  The  most  interesting  of 
the  three  parameters  is  the  shape  parameter,  whose  magnitude  determines  the 
nature  of  the  "conditional  response  rate,"  that  is,  the  conditional 
probability  that  an  examinee  who  has  not  responded  to  an  item  up  to  time  t 
will  respond  to  it  within  an  infinitesimally  short  time  interval  thereafter. 
A  brief  note  on  the  mathematical  and  conceptual  backgrounds  of  the  Weibull 
distribution,  introduced  in  our  previous  study,  will  be  described  in  the 
following  section. 

As  a  follow-up  to  the  Tatsuoka-Birenbaum  study,  Weibull 
distributions  were  fitted  to  every  item  in  the  posttest.  The  Weibull  fit  of 


almost  .ill  lt«*ms — Is  Items  on  addition  whieh  were  t.  night  prior  to  t  he 
students'  exposure  to  the  1’l.ATO  lessons — was  quite  poor  when  the  t  ltt  lug 
vijis  done  for  the  total  sample.  However,  the  separate  tits  ltt  two  groups 
that  had  been  earlier  Identified  as  having  dlstinetlv  different 
Inst  rue  t  Iona  l  backgrounds  were  very  good  tor  all  l  •*  Items  tsee  Appendix  l-a, 

1 -b ,  and  1-el.  Further,  It  was  found  that  the  value  of  shape  parameter  e 
differed  considerably  In  the  two  groups  tor  each  item,  being  higher  In 
one  group  t or  some  Items  and  lower  tor  others.  That  Is,  there  was  a  task 
bv  Instructional  method  Interaction  ettect  on  the  shape  parameter  c. 

The  foregoing  suggests  that  the  Welbull  shape  parameter  can  give  us 
a  handle  on  Identifying  Items  that  are  sensitive  to  particular  Information 
processing  skills.  After  Identifying  and  constructing  such  discriminating 
Items,  It  was  anticipated  that  an  Index  known  as  "person  conditional 
response  rate"  (TURK!,  to  be  developed  below,  could  be  used  tot 
"postdieting"  the  Instructional  background  of  students  and  routing  them 
accord  1 ng 1 v . 

KAVlONAl.l  OF  WFlRUl.l.  11 1  $TR1  RUTIONS 

Measuring  the  time  needed  to  achieve  a  g l von  goal  (that  Is,  response 
t  Intel  Is  easy  when  we  use  c  output  er-managed  testing,  but  since  It  is 
Impossible  to  collect  accurate  time  data  ltt  paper-and-penc 1 1  testing,  the 
latter  has  never  been  utilized  so  far  In  the  realm  ot  practical  application  ot 
psvehomet  r  Ics.  Tatsuoka  and  Tat  sucks  tl^S!  have  studied  the  statistical 
aspects  ot  response  time  distributions  and  their  charnct or  1st Ics  associated 
w  1 1  h  t  est- 1 1  ems . 

There  are  a  number  ot  theoretical  distributions  bv  which  the 
response  t  tme  data  mav  seem  t o  be  t  It  ted  well,  so  It  Is  necessary  tv'  follow 


some  guide  linos  as  to  what  sort  of  distribution  might  bo  appropriate  to 
represent  a  set  of  response  times  for  a  given  Item.  Kaseh  (14t>01  used  the 
two-parameter  gamma  distribution  as  a  model  for  the  time  taken  to  read  a 
passage  of  N  words.  He  used  the  Poisson  process  as  a  guide  to  his  model. 
The  occurrence  ot  a  response  is  a  random  event,  and  all  the  random  events 
are  assumed  to  be  of  the  same  kind.  He  was  interested  in  their  total 
number . 

Sato  and  others  ll975>  Introduced  the  Welbull  distribution,  which 
has  been  used  extensively  in  the  context  of  system-reliability  theory. 
Reliability  theory  is  the  study  of  the  probability  of  failure,  within  a 
given  t ime  span,  of  a  mechanical  or  electronic  system  as  a  fund  ion  ot  the 
probabilities  of  failure  ot  individual  components  of  the  svstem.  Their 
rationale  for  diverting  a  distribution  from  such  an  alien  field  is  that 
the  test  Item  is  identified  with  the  system  whose  "longevity"  is  being 
assessed;  the  student's  "attacks  on  the  Item"  correspond  to  the  shocks  or 
wear  and  tear  to  which  the  system  is  subjected,  and  the  eventual  solution 
of  the  item  Is  the  failure  ot  the  system.  It  is  plausible  to  Imagine  the 
student  as  being  Intent  on  "cracking  the  system"  by  answering  the  item 
correctly.  The  time  he/she  takes  in  doing  so — the  response  time — 
corresponds  to  the  "survival  time"  of  the  system.  This  rationale  for  the 
applicability  of  Welbull  distributions  for  item  response  time  does  not  lead 
to  a  derivation  of  the  distribution  or  density  function.  Mann  and  others 
(19741  have  said  the  distribution  was  empirically  discovered  rather  than 
deductively  derived  in  the  first  place.  Rut  later,  a  logical  basis  was 
postulated  as  an  ex  po_st_  facto  rationalization,  and  It  added  greatly  to  the 
credibility  of  the  distribution  in  the  theory  of  system  reliability.  This 


is  the  concept  of  hazard  rates,  which  is  essentially  the  conditional 
probability  that  a  system  which  has  survived  through  time  t  will  fail 
during  an  infinitesimal  time  interval  immediately  after  that. 
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We  introduced  a  similar  concept  "conditional  response  rate"  (CRK) 

in  our  study  as  a  logical  basis.  Suppose  f(t)  is  the  probability  density 

that  a  person  randomly  selected  from  the  population  will  respond  to  a  given 

item  during  the  Interval  [t,  t+dt].  Then,  the  proportion  of  individuals 

who  will  have  responded  to  the  item  by  time  t  is  the  probability 

distribution  function  F(t)  =  fl  f(u)du.  The  proportion  of  individuals 

'  to 

have  not  responded  to  the  item  by  time  t  is  1  -  F(t).  Consequently,  the 
conditional  probability  density  that  a  person  will  respond  to  the  item 
during  the  interval  [t,  t+dt]  given  that  he  or  she  has  not  responded  to  the 
item  up  to  time  t  is  given  by  f (t ) / [ 1-F (t ) ] . 

By  assuming  CRR  as  a  function  of  time  t  to  be  monotonically 
increasing,  or  decreasing,  as  a  power  function  of  t,  the  Weibull 
distribution  and  density  functions  can  be  exprssed  as  follows: 


(1)  F(t)  =  \  1  "  exP  I'  (-^°)C] 

l  0  u 


(2) 


f(t) 


t-t  „  . 

/  O.c-1 

j!(~) 

l  0 


t-t 


exp  [  -  (- 


Vi 


where  c(>0)  is  the  shape  parameter,  u(>0)  is  the  scale  parameter,  and 
to(>0)  is  the  location  parameter. 

If  £  =  1,  then  f(t)  is  a  negative  exponential  density  function.  If 
£  is  less  than  1,  then  f(t)  is  monotonically  decreasing  function.  The 
Weibull  density  function  is  symmetric  when  £  is  about  3.6.  Figure  2  is  a 
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copy  of  the  conditional  response  rate  function  obtained  from  live  data. 

The  increasing  curve  is  the  CRR  when  £  is  larger  than  1,  and  the  decreasing 
dot-graph  can  be  obtained  from  the  distribution  when  £  is  less  than  1. 

When  £  =  1,  CRR  becomes  a  straight  line  which  is  parallel  to  the  time  axis. 

Figures  3  and  4  show  the  displays  of  "goodness  of  fit"  tests 
with  the  normal  and  Weibull  distributions.  The  step  function  represents 
the  cumulative  distribution  of  a  set  of  response  times  to  an  item  which 
asks  about  matrix  multiplication.  The  continuous  line  stands  for  the 
estimated  theoretical  distribution  function.  The  Weibull  distribution 
fits  the  data  better  than  does  the  normal  distribution.  About  700  cases 
of  the  "goodness  of  fit"  test  were  carried  out  and  most  data  fitted 
either  the  Weibull  or  the  three  parameter  gamma  distributions. 

Theoretical  distributions  were  fitted  to  the  observed  response  time 
distribution  of  each  item  in  two  ways:  first  for  the  subgroup  of  students 
who  answered  the  item  correctly  (OK  subgroup),  and  second  for  the  subgroup 
of  students  who  answered  the  item  incorrectly  (NO  subgroup).  The  OK 
subgroup  and  NO  subgroup  had  considerably  different  estimated  Weibull 
parameters  but  both  showed  very  good  fits  for  most  items.  Figure  5  shows 
the  estimated  Weibull  distributions  of  the  OK  subgroup  and  NO  subgroup  for 
an  item  in  the  pretest  which  required  matrix  multiplication. 

The  Weibull  parameter  £  of  the  OK  subgroup  In  a  48-item  matrix 
algebra  pretest  correlates  with  the  numbers  of  options  in  the  item  (.32) 
and  the  difficulty  indices  (.41).  The  items  with  more  choice  options 
tended  to  have  large  c-values.  If  our  Interpretation,  made  in  Tatsuoka  4 
Tatsuoka  (ll>78),  that  the  item  c-value  reflects  the  degree  of  engagement 
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gure  4  Goodness  of  fit  test  for  the  response 
noraal  distribution 
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1 A 

students  show  when  the  Item  Is  correct,  we  may  conclude  that  within  the 
range  represented,  the  larger  the  number  of  options,  the  greater  the 
engagement  students  feel.  This  seems  reasonable  since  items  with  more 
options  present  more  of  a  cognitive  task  and  hence  probably  induce 
greater  Involvement  on  the  part  of  students.  About  ten  items  in  the  test 
asking  mathematical  properties  of  orthogonal  transformations,  eigen¬ 
values  and  -vectors,  were  very  difficult  for  many  students  in  the 
course.  These  items  tended  to  have  the  smaller  Weibull  shape  parameters 
c  in  both  OK  and  NO  subgroups.  A  similar  observation  was  obtained  from 
the  b4-ltem  signed-number  pretest. 

The  three  parameter  gamma  distributions  fit  well  the  items  which 
repeatedly  require  a  simple  mechanical  task,  while  the  Weibull 
distributions  fit  wo  1 1  the  items  which  require  a  higher  cognitive  task  to 
respond  to.  Since  the  CRR  of  the  gamma  distributions  is  always  non- 
decreasing,  that  is,  either  moncton leal lv  Increasing  or  parallel  to  the  time 
axis  (see  Appendix  d } ,  the  interpretation  of  the  Weibull  shape  parameter 
(see  Figure  2)  provides  wider  applicabilty  than  the  gamma  shape  parameter 
does.  Moreover,  the  parameter  estimation  routine  by  maximum  likelihood 
usually  falls  to  give  us  convergent  estimated  gamma  parameters  when  items 
have  decreasing  CRRs. 

LATENT  RES PONS K  TIME  MODEL 

Latent  Response  lime  Variable  and  Item  Response  Time  Characteristic  Carve 

As  a  first  step  toward  developing  the  person  conditional  response 
rate  (I’CRR),  we  postulate  the  existence  of  a  latent  response  time  variable. 
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(t),  analogous  to  the  ability  variable  a  In  latent  trait  theory.  Thus, 
given  a  set  of  ri  items,  the  performance  on  which  is  affected  by  9,  we 
assume  that  there  also  exists  a  variable  which  affects  the  time  taken  by  an 
examinee  to  answer  each  of  these  items.  We  shall  not  attempt  to  give  any 
precise  psychological  meaning  to  this  construct  beyond  saying  that  it  may 
be  regarded  as  a  pervasive  trait  of  individuals  to  be  slow  or  quick  in 
solving  items  in  a  certain  domain. 

The  plausibility  of  this  postulation  is  suggested  by  the  following 
empirical  findings.  In  the  Tatsuoka  and  Tatsuoka  study  (1978),  the 
performance  scores  on  a  48-item  matrix  algebra  test  were  found  to  have  a 
strong  tendency  toward  unidimensionality .  At  the  same  time,  the  response 
times  for  these  items  showed  a  suggestion  of  unidimensionality  by  the  scree 
test.  On  the  other  hand,  the  posttest  for  the  signed-number  lessons 
mentioned  earlier  showed  no  semblance  of  unidimensionality  in  the  total 
sample.  However,  when  one  instructional-background  group  (we  will  call 
this  Group  2  hereafter)  identified  by  cluster  analysis  was  removed,  both 
performance  scores  and  response  times  came  somewhat  closer  to  being 
unidimensional  in  the  remaining  sample. 

On  the  strength  of  these  observations  and  of  the  fact,  mentioned 
earlier,  that  the  Weibull  distribution  fits  the  response-time  data  for  most 
items,  we  develop  a  model  for  item  response  time  in  the  following  manner, 
roughly  paralleling  latent  trait  theory, 
bet 


(3) 
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if? 
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i. 


lb 


be  the  deviation  of  Individual  i's  response  time  t^g  for  item  g  from  his 
or  her  mean  response  time  tj.  over  the  given  set  of  n  items.  Now  t 
be  conceptualized  as  the  expectation  E(t^g)  of  that  person's  response  times 
over  an  infinite  number  of  items  of  the  same  type  as  those  in  our  set  of  n. 
Then  ^  +  d|g  is  approximately  equal  to  tjg,  so  if  we  let  1  vary  across  the 
population,  it  is  reasonable  to  assume  that  +  dig  follows  a  Weibull 
distribution  just  as  t^g  does.  We  therefore  define 


(4)  P.  (t)  =  1  -  exp  [-  . _ g] 

‘*5  1  u  "  ’ 
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as  the  "response-time  characteristic  function"  (RTCF)  for  item  g,  where  we 
have  taken  tQ  =  0  in  the  general  expression  (2)  for  the  Weibull 
distribution  function  to  simplify  the  task  of  parameter  estimation.  This 
is  interpreted  to  represent  the  probability  that  a  person  whose  latent 
response  Lime  is  t  will  arrive  at  the  answer  to  item  g  at  or  prior  to 
t  ime  t  +  d„  • 

O 

For  estimating  the  two  item  parameters  c„  and  u  as  well  as  the 

O  P 

person  parameter  T^,  the  density  function  corresponding  to  (4)  is 
written  in  accordance  with  Equation  (1)  (with  tQ  set  equal  to  0)  for  each 
person  i,  and  the  product  over  all  items  and  those  individuals  who  got 
the  item  correct  is  formed  to  obtain  the  likelihood  function.  That  is. 
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where  Ng  is  the  number  of  subjects  in  OK-Subgroup  for  item  g. 

Before  going  to  the  next  step  of  developing  the  PCRR  function 


G(  t),  let  us  see  how  t  itself,  once  estimated,  can  help  in  the  task  of 


postdLcting  a  student's  instructional  background.  Suppose  there  are  two 
Items  that  differentiate  between  two  prior  instructions  A  and  B  by 
actually  showing  a  reversal  in  the  magnitude-order  of  mean  times  required 
for  their  solution  by  examinees  who  were  previously  taught  by  these  two 
methods.  Table  1  shows  the  mean  response  time  (also  with  the  estimates  of 
Weibull  parameter  £  and  CRK)  of  12  items  described  earlier  for  the  two 
groups,  of  which  prior  instructional  methods  are  A  and  B  respectively.  Let 

us  take  the  means  of  items  32  and  33  as  an  example,  then 

t1A  =  9.65  sec,  tiR  “  5.43  sec. 

£2A  “  4.62  sec.,  t^g  “  7.22  sec. 

Given  these  data  and  the  observed  response  times  tj  and  ti  for 
the  two  Items  of  a  person  about  whom  we  have  no  other  Information,  a 
natural  but  simple-minded  decision  rule  for  postdieting  his/her  instruct  Iona 
background  would  be  to  choose  A  if  tj  >  tT  and  B,  otherwise.  The  trouble, 
of  course,  is  that  the  magnitude-order  of  the  two  observed  times  could  be 
reversed  from  the  "true"  order  by  errors  of  measurement.  Knowledge  of  the 
person's  r  j  may  help  increase  our  confidence  in  our  postdiction,  using  the 
following  sequential  decision  rule — again  deliberately  a  simple-minded  one. 

We  first  administer  only  Item  1  to  this  person.  Now  suppose 
his/her  tj  is  less  than  6.73  sec.  (the  mean  of  the  four  mean  response 
time  listed  above).  Then,  If  tj>  9.65,  we  choose  A,  and  the  testing  is 
terminated.  If,  on  the  other  hand,  tj<  9.65,  we  further  administer  item 
2,  and  choose  B  iff  t2  >  7.22,  and  otherwise  choose  A  or  R  according  as 
t  j  >t  2  or  tl<t2*  respectively.  When  the  person's  is  greater  than  or 
equal  to  6.73,  the  sequential  decision  will  be  the  dual  of  the  above. 
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Table  1 

The  Means  of  Response  Time,  Observed  Conditional  Response  Rate  at  Mean 
and  Weibull  Shape  Parameters  of  Addition  Problems  of 
64-item  Signed  Number  Test 


Item  Mean  CRR  at  Mean  Shape  Parameter 
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FIGURE  ^  Sequential  decision  for  Postdating  Method  A  or  B  based  on 
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Namely,  It  tj  <  5.43,  we  choose  B;  If  tj  >  5.43  we  further  administer  item 
2,  and  choose  A  iff  t  i <  4.t>2.  It  t  >  >  4.t>2  we  choose  A  or  B  according  to 
the  magnitude-order  of  tj  and  t>. 

The  above  discussion  was  kept  at  a  simple,  intuitive  level  to 
highlight  the  meaning  of  and  how  It  would  affect  our  Interpretation  ot 
an  observed  response  time.  Refinements  would  Include  getting  conditional 
response-rate  distributions  for  each  Item  given  instruction  background  and 
the  value  of  i  .  Further,  with  a  suitable  assumption  concerning  the 
distribution  of  *,  one  could  derive  the  posterior  probability  for  each 
instructional  background  given  1  and  the  magnitude  for  each  order  of  tj 
and  t^*  With  more  than  two  instructional  backgrounds  and  a  larger  number 
of  dlscrimlnat ing  items,  the  magnitude  order  of  two  response  times  would  be 
generalized  to  a  vector  of  response  times  exhibiting  different  patterns, 
l.e.,  permutation  of  the  magnitudes  of  the  elements. 

Person  Conditional  Response  Rate _ tJVRR) 

l.et  me  backtrack  a  little  bit  and  explain  why  the  Welbull  family 

was  chosen  over  the  gamma  despite  the  latter's  having  a  longer  tradition  of 

usage  in  response-time  mode  1 s  (Rasch,  ldt>0;  Restle  A  Oav  l  s ,  ldt>2,  among 
others).  First,  the  gamma  distributions  are  Indicated  when  distinct  stages 
are  identifiable  In  the  process  of  solving  the  tasks.  In  which  case  c  must 
be  an  Integer  representing  the  number  ot  stages.  Second,  the  shape 

parameter  c  of  the  Welbull  family  has  the  Interesting  feature  ot  apparently 

distinguishing  between  different  Informat  ion-process ing  skills  associated 
with  different  instruct  Iona l  backgrounds.  Tills  feature  is  no  doubt  related 
to  the  fact  that  the  magnitude  ot  c  (l.e.,  whether  c  Is  greater  than,  equal 


to,  or  less  than  1)  determines  the  nature  of  the  item  conditional  response- 
rate  function  (ICRR),  which  describes  whether  perseverance  increases  the 
chances  of  an  examinee's  responding  to  an  item,  responses  occur  at  random 
times,  or  a  point  of  diminishing  returns  is  reached  early.  In  other  words, 
it  can  be  said  as  mentioned  In  the  first  section  on  "Rationale  of  Welbull 
Distributions,"  that  £  is  sensitive  to  the  degree  of  involvement  students 
show.  Two  different  instructional  methods  usually  require  different  steps 
of  information-processing  skills,  thus  each  method  requires  a  different 
degree  of  Involvement  In  solving  a  given  item.  For  example,  some  items  in 
Table  1,  such  as  "10+”11  =  ?"  in  the  signed-tu  mber  posttest,  yield  not  only 
different  values  of  c,  but  also  significantly  different  mean  response-times 
depending  on  whether  the  sequential  or  numbei — lines  method  is  used  for 
answering  it,  as  dictated  by  the  examinee's  instructional  background. 
Moreover,  the  convenient  ICRR  function  is  readily  expressed  in  closed  form 
for  a  Welbull  distribution,  but  cannot  be  so  expressed  for  a  gamma 
distribution  because  the  incomplete  gamma  function  cannot  be  expressed 
analvt leal ly  • 

We  first  present  the  ICRR  function,  l.e.,  the  probability  that  an 
examinee  who  has  not  responded  to  an  Item  by  time  t_  will  do  so  within  an 
infinitesimal  time  interval  thereafter.  When  item  response  times  follow  a 
Welbull  distribution,  this  function  H„(t)  is  given  by  f„(t)/[l-  F  (t)J, 

f*  f*  r' 

where  f  (t)  and  F  (t)  are  expressions  (l)  and  (21  with  the  parameters 

r*  r 

subscripted  with  a  g  for  item  g  and,  in  our  case,  t^ 


(b)  Hg(t) 


c 


t 


set  equal  to  0.  Hence 
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From  this  the  transition  to  PCRR  is  made  in  a  manner  analogous  to 
going  from  an  item  characteristic  curve  to  a  person  characteristic  curve, 
first  suggested  by  Mosier  (1936),  recently  by  Weiss  in  1973,  and  discussed 
in  greater  detail  by  Lumsden  in  1978.  In  their  studies  a  plot  is  made  for 
each  individual  of  the  proportions  of  items  of  varying  difficulty 
(represented  by  the  horizontal  axis)  which  are  passed  by  that  individual. 

In  our  case,  the  ordinate  at  each  point  along  the  horizontal  axis 
representing  the  mean  response  time  of  an  item  would  be  the  value  of  Hg(  t )- 
where  x  is  the  latent  response  time  for  the  particular  person — computed 
from  (6)  with  the  parameter  values  proper  to  that  item  substituted  for  Ug 
and  Cg.  Note  that  when  shape  parameter  c^  equals  unity  for  all  items,  the 
PCCR  curves  are  identical  for  all  persons.  Thus,  utilizing  the  negative 
exponential  distribution  (i.e.,  a  special  case  of  the  Weibull  distribution 
functions)  for  our  purpose  will  not  work. 

Equation  (6)  defines  a  function  whose  curve  characterizes 
behavior  of  item  g  over  time  in  terms  of  the  probability  of  reaching  an 
answer.  The  steeper  the  slope  of  a  curve  is,  the  greater  the  chance 
that  item  g  will  be  solved  as  time  goes  by.  The  steepness  of  curves  is 
a  characteristic  attributed  to  a  given  item,  similar  to  item  discrimination 
index  in  latent  trait  theory.  We  can  define  a  sort  of  conditional  response 
rate  function  on  variable  x  as  follows: 


For  a  fixed  person  i. 


(8)  C.(Hd  )  =  Hj  (t.),  g  =  1 . n. 

1 R  R 

It  should  be  noted  that  the  T  in  Equations  (7)  and  (8)  is  merely  an 
arbitrary  time  point  and  bears  no  relation  to  a  person's  latent  response¬ 
time  (except  for  coinciding  in  numerical  value).  Only  in  the  context  of 
the  random  variable  x  +  dg  does  have  the  sense  of  latent  response  time, 
but  to  use  r  +  d^  as  the  argument  of  (•)  would  be  meaningless  because 
t  +  dg  is  approximately  the  person's  observed  response  time  for  item  g, 
and  it  would  be  a  contradiction  in  terms  to  speak  of  the  person's 
responding  in  the  next  moment  given  that  he/she  has  not  responded  up  to  the 
actual  time  point  at  which  he/she  did  respond. 

The  above  remarks  Indicate  that  the  particular  approach  attempted  here 
for  defining  PCRR  was  futile,  but  not  that  the  concept  of  PCRR  itself  is 
meaningless.  An  alternative,  more  justifiable  approach  might  be  to 
transform  response  time  to  an  approximate  normal  variable.  The 
transformation  we  derived  by  the  usual  method  of  obtaining  variance- 
stabilizing  transformations  was  unusable  because  It  was  an  arcsine 
transformation  whose  argument  could  exceed  one. *  therefore  extended  the 

usual  method  by  taking  up  to  the  second  term.  Instead  of  only  the  first,  in 
the  Taylor  series  expansion  on  which  the  transformation  is  based.  The 
result  was  that  the  transform  y  is  the  solution  of  the  following,  rather 
formidable  differential  equation: 


*This  fact  was  noticed  and  pointed  out  to  us  by  Jim  Paulson  at 
the  conference. 
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Where 

(9)  h(t)(y")~  +  g(m)y"y'  +  f(m)y'  =  C, 

where 

h(t)  =  uo5|r(l+4/c)  -  r“  ( 1+2/ c)  |  /  4  -  [Mo3r(l+3/c)(t-to)  -  2uo2!’(1+2/c)  (t-t^T 
+  (t-to)41 

g(t)  =  V  3r( 1+3/c)  -  3p  2r ( 1+2/e) (t-t  )  +  2 ( t-t  ) 
o  o  o  o 

£(t)  =  U  2r(i+2/c)  -  (t-t  )J 

o  o 

and  c_  is  an  arbitrary  positive  constant.  (We  have  not  solves  this  yet  but  a 
mathematician  colleague  assures  us  that  It  is  soluble!)  If  we  are  fur ther 
willing  to  assume  that  r  is  normallly  distributed  (which  seems  reasonable 
by  virtue  of  the  central  limit  theorem,  since  t  is  a  person's  mean 
response  time  over  an  Infinite  set  of  items  which  may  be  regarded  as 
exhibiting  local  independence  if  unidimensionality  holds),  then  y  and  r 
would  jointly  follow  a  bivariate  normal  distribution.  Hence,  if  their 
correlation, p,  can  be  be  estimated  (roughly  analogous  to  communality 
estimation  in  factor  analysis),  the  joint  distribution  would  be  uniquely 
determined.  From  tills  and  the  distribution  of T  ,  the  conditional 
distribution  of  y  given  can  be  determined.  All  quantities  associated  with 
per?  ns  having  a  particular  T-value  are  computed  from  this  conditional 
d  1st  r  l but  ion. 

ESTIMATION  OF  THE  PARAMETERS 

We  are  now  Interested  in  estimating  c„  ,tig  and  i^,  g-l,...,n, 

1-1,..., N  simultaneously.  We  have  to  choose  the  set  of  admissible 

values  of  these  parameters  which  makes  the  log-likelihood  function,  Inh,  the 


maximum.  Unlike  the  case  of  dealing  will)  performance  scores,  response 
time  represents  two  different  cases;  one  is  a  group  whose  members  reached 
the  correct  answer,  while  the  other  group  consists  of  those  ending  up  with 
wrong  answers.  Response  time  in  the  OK-subgroup  means  the  time  needed  to 
attain  a  given  goal  using  a  successful  information-processing  skill  (or 
skills),  but  it  is  not  that  simple  with  the  NO-subgroup.  A  brief 
investigation  of  error  analysis  for  the  NO-subgroup  Indicates  that  various 
kinds  of  misconceptions  at  different  progressive  stages  of  reaching  the 
correct  answer  for  a  given  item  might  have  occurred.  Therefore,  we  will 
consider  only  the  OK-subgroup  for  our  purpose  in  this  paper. 

Differentiating  the  logarithm  of  Equation  (5)  by  parameters  c^ 
and  Ug  respectively,  and  setting  the  results  equal  to  zero,  we  obtain 

the  following  simultaneous  equations. 
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The  maximum  likelihood  method  using  the  Newton— Raphson  Iteration 
procedure  provides  estimates  of  the  roots  of  Equations  (10)  and  (11) 
where,  l-l,...,N  are  substituted  by  the  mean  response  t Ime  of  each 


person  over  items,  g=l,...,n. 


(1 


Then,  t  ho  roots  ot  liquation  (  l .! )  alter  the  nowly  estimated  e 
anil  u^  values  arc  subs 1 1 tut  ml  are  sought  by  the  same  procedure. 

A  suit  lclent  but  not  necessary  condition  that  any  ol  these 
stationary  values  u,  e,  be  local  maxima  Is  that 
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It  should  be  noted  that  lucquallt  les  (I  ))  and  (IS)  are  always  negat  lv«' 
but  Inequal  ttv  (14)  will  be  negative  only  when  the  estimates  ol  u  are 
close  enough  to  be  the  roots  ol  Kquat  Ion  (11).  In  some  earlier  stages 
of  Iterations,  the  condition  lor  u^,  to  yield  a  local  maximum  might  not  be 
sat  1st  led.  Thus,  it  Is  Important  to  select  appropriate  statt  Ing  values 
lor  the  estimates  of  u^.s. 

Kor  est  Imat  Ion  ol  i  ,  we  sock  solut  Ions  ol  Kquat  Ion  (1.’) 
for  which  Inequality  (IM  its  satisfied.  11  o,  -  I  lor  all  g,  g»l,...,u, 
then  Kquat  Ion  (l.’)  becomes 
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Since  scale  parameter  u^,  equals  the  mean  ot  observed  response  t  l mo 
when  shape  parameter  e  Its  equal  to  unltv,  this  equation  becomes 
equivalent  to 
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The  reciprocal  of  observed  mean  cannot  be  zero  for  any  item  g. 

Therefor^  there  should  be  some  g  such  that  c^,  ¥  1  •  This  implies  that  the 
maximum  likelihood  method  does  not  work  for  response  time  models  associated 
with  the  negative  exponential  distributions  as  long  as  the  models  are 
formulated  assuming  unidimensionality.  Moreover,  the  notion  of  "person 
conditional  response  rate"  which  is  parallel  to  that  of  the  person's 
characteristic  curve  will  not  be  applicable  to  these  models.  This  is 
because  conditional  response  rate  functions  are  always  parallel  to  the 
horizontal  axis  when  the  occurrence  of  a  response  is  a  random  event  and  all 
random  events  are  assumed  to  be  of  the  same  kind,  which  is  the  case  of 
negative  exponential  distributions. 

Numerical  F.xamp  1  e 

A  computer  program  for  estimating  parameters  u0,  c„  and  T  for 

o  o  i 

g-l,...,n,  i=l,...,N  was  written  on  the  PLATO  system  by  Robert  Bail  lie. 

The  parameters  in  our  model  were  successfully  estimated  for  some  sample 
data,  the  pretest  data  from  signed-number  arithmetic  lessons. 

Unfortunately,  the  posttest  data  of  signed-number  operations  described 
in  "Introduction"  and  elsewhere  (see  also  Appendix  l-a,  I-b,  and  I-c)  could 
not  provide  stable  estimates  with  this  computer  program  because  of  both 
small  sample  size  for  Group  2  and  too  few  items — only  the  12  items  that 
were  of  most  interest  to  us.  When  the  observed  response- 
t lme  data  were  fitted  to  the  Weibull  distributions  before,  it  was 
observed  that  the  items  testing  the  same  skill  in  the  pretest  showed  a 
systematic  change  with  the  estimates  of  u  and  tc  according  to  their 
order  of  presentation,  even  though  the  difficulties  of  these  parallel 
Items  did  not  show  any  noticeable  change. 


With  this  new  model,  the  changes  in  the  slopes  of  the  parallel 
items  have  a  strong  tendency  of  being  monotonically  increasing.  For  example 
Items  3  and  45  ask  "-3+2=?"  and  "-7+5=?",  respectively.  The  dotted  lines 
in  Figure  7  are  CRR  functions  associated  with  the  observed  response-time, 
while  the  solid  lines  are  the  theoretically  derived  CRR  functions.  It  is 
interesting  to  note  that  the  random  variable  +  dlg  can  be  rewritten  as 

( Ti  ~  *1 • )  +  t^g  =  T|g.  If  we  denote  -  t^.  =  -  then  T^g  can  be 
expressed  as  tig  -  .  Hence,  the  observed  response-time  t^g  becomes  the 

sum  of  a  t rue- score- like  T^g  and  an  "error"  t  Therefore, 

it  might  be  considered  that  the  theoretical  CRR  functions  are  defined  on 

a  sort  of  true-score  random  variable,  T^g. 

SUMMARY  AND  DISCUSSION 

The  customary  method  for  assigning  a  score  to  an  individual  on 
adaptive  tests — or,  for  that  matter,  whenever  a  latent  trait  model  is 
employed — is  to  use  the  estimate  0  of  the  ability  (or  achievement) 
parameter.  This  may  be  adequate  when  the  only  purpose  of  testing  is  to 
calibrate  the  individual's  ability  or  achievement  level.  When  the 
further  purpose  of  using  0  as  the  basis  for  routing  the  student  to  a 
suitable  starting  point  in  a  lesson  series  is  involved,  however,  sole 
reliance  on  0  can  create  serious  problems.  This  Is  because  two 
examinees  may  have  identical  response  patterns  (and  hence  0  values)  and 
yet  differ  drastically  in  the  manners  in  which  they  arrived  at  their 
answers  to  the  items,  right  or  wrong  as  the  case  may  be.  By  "the  manners" 
we  mean  the  cognitive  processes  and  information-processing  skills  that  are 
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brought  into  play.  Efficient  and  effective  routing  of  students  to  lessons 
requires  this  "deeper  diagnosis"  instead  of  mere  information  as  to  which 
items  they  get  right  or  wrong. 

Increasing  recognition  is  being  given  to  this  fact,  as  evidenced 
by  the  number  of  studies  either  directly  or  indirectly  germane  to  it 
that  have  recently  been  done  by  cognitive  psychologists  (e.g.,  Anderson 
£t  al. ,  1978;  Carroll,  1978;  Frederiksen,  1978;  Greeno,  1977;  Groen  & 

Perkum;  1972;  Heller  &  Greeno,  1977;  Rose,  1977;  Sternberg,  1979).  These 
studies  have  demonstrated  the  existence  of  a  variety  of  cognitive  processes 
which  differ  from  individual  to  individual. 

One  clue  to  the  type  of  cognitive  process  employed  by  a  student 
in  solving  a  given  problem  can  come  from  knowing  his/her  instructional 
background.  Fortunately,  a  follow-up  study  of  Tatsuoka  and  Birenbaum 
(1979)  indicates  that  the  Weibull  shape  parameter  c^,  obtained  by 
fitting  response-time  data,  is  helpful  in  differentiating  among  various 
instructional  methods  associated  with  signed-number  operations.  The 
Weibull  distributions  can  be  mathematically  derived  from  the  assumptions 
that  conditional  response  rate  (C-RR) — essentially  the  conditional 
probability  that  a  person  will  respond  to  a  given  item  during  the  interval 
[t,t+dt]  given  that  he/she  has  not  responded  to  the  item  up  to  time  t — is 
monotonical ly  increasing,  decreasing  or  constant.  The  slope  of  the 
conditional  response  rate  function  for  a  given  item  is  determined  by  the 
magnitude  of  the  shape  parameter  and  the  mean  of  the  item  response-time.  If 
c_  is  larger  than  1,  then  CRR  is  a  monotonically  increasing  function.  If 
c  =  1,  then  CRR  is  constant.  Some  types  of  information-processing  skills 
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require  a  greater  amount  of  involvement  In  the  student's  effort  in  solving  a 
given  problem,  while  others  don't  require  so  much  to  get  the  answer  to 
the  same  item.  The  magnitude  of  shape  parameter  £  and  mean  response- 
time  for  the  former  become  noticeably  larger  than  those  in  the  latter 
case.  Therefore  the  slopes  of  CRR  functions  differ  in  steepness  to  a 
greater  extent.  This  sensitivity  of  the  Weibull  distributions  to  the 
procedures  associated  with  different  teaching  methods  is  an  advantage  in 
dealing  with  psychological  research,  as  Scheiblechner  (1979)  states: 

"The  exponential  or  Weibull  distribution  is  an  adequate  model  for  more 
sorts  of  psychological  data  than  Is  commonly  assumed  If  the  parametelc 
structure  of  the  latencies  Is  properly  chosen." 

First,  we  assume  that  for  a  given  set  of  items  there  exists  a 
latent  variable  which  affects  the  time  taken  by  an  examinee  to  answer 
each  of  these  items.  We  formulated  a  model  associated  with  response¬ 
time,  roughly  paralleling  latent  trait  theory,  on  the  strength  of  the 
observed  fact  that  the  Weibull  distribution  fits  the  response-time  data 
for  most  Items.  Our  main  concern  In  the  model  is  to  express  the 
relationship  between  latent  response-time  variable  and  information- 
processing  skills. 

An  estimation  routine  of  the  parameters  by  the  maximum  likelihood 
method  was  programmed  by  Robert  Ball  lie  on  the  F1.AT0  system  and  a  numerical 
example  was  shown  (see  Figure  7).  The  maximum  likelihood  method  is  not 
applicable  to  estimate  Weibull  parameters  when  all  shape  parameters  are 
supposed  to  be  1,  that  Is  the  case  of  negative  exponential  distributions. 
Further  research  will  be  necessary  in  exploring  a  different  parameter 
estimation  procedure,  such  as  the  conditional  maximum  likelihood  method. 
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Information  function  of  item  g,  Ig(  r  )  was  Integrated 
numerically  and  found  to  be  always  constant  except  for  c~  *  1 .  However, 

o 

we  relegate  Its  discussion  to  the  next  technical  report. 

The  particular  approach  attempted  here  for  defining  item  CRR  and 
Person  Conditional  Response  Rate  functions  resulted  in  the  loss  of  the 
attractive  feature  of  being  capable  of  providing  mathemat ical  meaning  to 
the  curves  in  terms  of  .  However,  the  attractive  feature  still  holds 

for  the  variab'i  mentioned  in  the  numerical  example,  that  is,  T^.  An 
alternative  approach  was  outlined,  but  further  research  is  necessary  to 
make  this  approach  operational. 
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Appendix  1-a 

Kolmogorov-Smlrnov  Tests  for  the  Total  Sample  of  the  Signed-Number  Test 
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Navy  Personnel  R&D  ('enter 
San  Diego,  CA  92152 


Personnel  &  Training  Research 
Programs  (Code  458) 

Office  of  Naval  Research 
Arlington,  VA  22217 

Psychologist 

Office  of  Naval  Research  Branch 
223  Old  Mary lebone  Road 
London,  NW,  15th  England 


Mr.  Arnold  Rubenstein 
Naval  Personnel  Support  Technology 
Naval  Material  Command  (08T244) 
Room  1044,  Crystal  Plaza  #5 
2221  Jefferson  Davis  Highway 
Arlington,  VA  20360 
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Dr.  Worth  Scanland 

Chief  of  Naval  Education  and  Training 
Code  N-5 

NAS,  Pensacola,  FL  32508 

A.  A.  Sjoholm 
Tech.  Support,  Code  201 
Navy  Personnel  R  and  D  Center 
San  Diego,  CA  92152 

Mr.  Robert  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987E 

Washington,  DC  20350 

Dr.  Alfred  F.  Smode 
Training  Analysis  and  Evaluation 
Group  (TAEG) 

Dept,  of  the  Navy 
Orlando,  FL  32813 

Dr.  Richard  Sorensen 

Navy  Personnel  R  and  D  Center 

San  Diego,  CA  92152 

CDR  Charles  J.  Theisen,  Jr.,  MSC, 

USN 

Head  Human  Factors  Engineering  Div. 
Naval  Air  Development  Center 
Warminster,  PA  18974 

W.  Gary  Thomson 

Naval  Oceans  Systems  Center 

Code  7132 

San  Diego,  CA  92152 

Dr.  Ronald  Weitzman 
Department  of  Adminstrat ive 
Sciences 

U.S.  Naval  Postgraduate  School 
Monterey,  CA  93940 

Dr.  Martin  F.  Wiskoff 

Navy  Personnel  R  and  D  Center 

San  Diego,  CA  92152 


Technical  Director 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

HQ  USAREUE  &  7th  Army 
ODCSOPS 

USAAREUE  Director  of  GED 
APO  New  York  09403 

LCOL  Gary  Bloedorn 

Training  Effectiveness  Analysis  Division 
US  Army  TRADOC  Systems  Analysis  Activity 
White  Sands  Missile  Range,  NM  88002 

Dr.  Ralph  Dusek 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Beatrice  J.  Farr 

Army  Research  Institute  (PERI-OK) 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Myron  Fischl 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Ed  Johnson 

U.  S.  Army  Research  Institute 
5001  Eisenhower  Blvd. 

Alexandria,  VA  22333 

Dr.  Michael  Kaplan 
U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Milton  S.  Katz 
Individual  Training  and  Skill 
Evaluation  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


Air  Force 


Dr.  Milt  Maier 

U.S.  Array  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Harold  F.  O'Neil,  Jr. 

Attn:  PERI-OK 

5001  Eisenhower  Avenue 

Alexandria,  VA  22333 

Dr.  Robert  Ross 

U.S.  Army  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Robert  Sasmor 

U.S.  Array  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Director,  Training  Development 
U.S.  Army  Administration  Center 
ATTN:  Dr.  Sherrill 
Ft.  Benjamin  Harrison,  IN  46218 

Dr.  Frederick  Steinheiser 
U.S.  Array  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Joseph  Ward 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


Air  Force  Human  Resources  Lab 
AFHRL/PED 

Brooks  AFB ,  TX  78235 

Air  University  Library 
AUL/LSF.  76/443 
Maxwell  AFB,  AL  36112 

Dr.  Philip  De  Leo 
AFHRL/TT 

Lowry  AFB,  CO  80230 

Dr.  G.  A.  Eckstrand 
AFHRL/AS 

Wright-Patterson  AFB,  OH  45433 

Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

CDR.  Mercer 
CNET  Liaison  Officer 
ARGRL/F lying  Training  Div. 
Williams  AFB,  AZ  85222 

Dr.  Ross  L.  Morgan  (AFHRL/ASR) 
Wright-Patterson  AFB 
Ohio  45433 

Dr.  Roger  Pennell 
AFHRL/TT 

Lowry  AFB,  Co  80230 

Personnel  Analysis  Division 
HQ  USAF/DPXXA 
Washington,  DC  20330 

Research  Branch 
AFMPC/DPMYP 

Randolph  AFB,  TX  78148 
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Dr.  Malcolm  Ree 
AFHRL /PED 

Brooks  AFB,  TX  78235 
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Dr.  Matty  Rockwav  (AFHRL/TT) 
Lowry  A  KB 
Colorado  80230 

Jack  A.  Thorpe,  Capt,  USAF 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB,  DC  20332 

Brian  K.  Waters,  LCOL,  USAF 
Air  Univerity 
Maxwell  AFB 
Montgomery,  AL  36112 


Marines 

H.  William  Greenup 
Education  Advisor  (E031) 
Education  Center,  MCDEC 
Quantico,  VA  22134 

Dr.  A.  L.  Slafkosky 
Scienfitic  Advisor  (Code  RD-1 ) 
HQ,  U.S.  Marine  Corps 
Washington,  D.C.  20380 
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Coast  Guard 

Mr.  Richard  Lanterman 
Psychological  Research  (G-P-l/62) 
U.S.  Coast  Guard  HQ 
Washington,  DC  20590 

Dr.  Thomas  Warm 
II. S.  Coast  Guard  Institute 
P.0.  Substation  18 
Oklahoma  City,  OK  73169 


Other  DoD 

Dr.  Stephan  Andriole 

Advanced  Research  Projects  Agency 

1400  Wilson  Blvd. 

Arlington,  VA  22209 

Defense  Documentation  Center 
Cameron  Station,  Bldg.  5 
Alexandria,  VA  22314 
Attn:  TC 

Dr.  Dexter  Fletcher 

Advanced  Research  Projects  Agency 

1400  Wilson  Blvd. 

Arlington,  VA  22209 

Dr.  William  Graham 
Testing  Directorate 
MEPCOM 

Ft.  Sheridan,  IL  60037 

Military  Assistant  for  Training  and 
Personnel  Technology 
Office  of  the  Under  Secretary  of 

Defense  for  Research  and  Engineering 
Room  3D129,  The  Pentagon 
Washington,  DC  20301 

Major  Wayne  Sellman,  USAF 
Office  of  the  Assistant  Secretary 
of  Defence  (MRA  and  L) 

3B930  The  Pentagon 
Washington,  DC  20301 
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Civil  Cove  rm:u » n  t 

Dr.  Susan  Chipnian 
Basic  Skills  Program 
National  Institute  of  Kduc.it  ion 
1200  19th  Street  NW 
Washington,  DC  20208 

Dr.  William  Clorliain,  Director 
Personnel  R&D  Center 
Office  of  Personnel  Managment 
1900  K  Street  NW 
Washington,  DC  204  IS 

Dr.  Joseph  I.  l.ipson 
Division  of  Science  Kduc.it  ion 
Room  W-h  38 

National  Science  Konnd.it  ion 
Washington,  DC  20SS0 

Dr.  John  Mays 

National  Institute  of  ('duration 
1200  19th  Street  NW 
Washington,  DC  20208 

Dr .  A r no  I d  Me  1 mrd 
National  Instltut*  of  Kduc.it  ion 
1  200  I'M  li  .  *  reet  NW 
•ashlngt  n,  Di  20208 
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Dr .  1'lioiiias  .  St  iciit 
Basic  Skills  Program 
National  Institute  of  Kduc.it  ion 
1200  19th  Street  NW 
Washington,  DC  20208 

Dr.  Ve  rn  W .  1'r  tv 
Personnel  K  and  D  Center 
V.S.  Civil  Service  Commission 
1900  K  Street  NW 
Washington,  DC  204  IS 

Dr.  Frank  Withrow 
O.S.  Office  of  Kdue.it  ion 
400  bth  Street  SW 
Washington,  DC  20202 

Dr.  Joseph  !..  Young,  Director 
Memory  and  Cognitive  Processes 
National  Science  Kound.it  ion 
Washington,  DC  20SS0 


>t  .  lalittia  P.  Sana  t  hnnaii 
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Non  Government 

Dr.  Earl  A.  Alluisl 
HQ,  AFHRL  (AFSC) 

Brooks  AFB,  TX  78235 

Dr.  Erling  B.  Anderson 

University  of  Copenhagen 

Studiestraedt 

Copenhagen 

Denmark 

Dr.  John  R.  Anderson 
Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

Dr.  Michael  Atwood 
Science  Applications  Institute 
40  Denver  Tech.  Center  West 
7935  E.  Prentice  Avenue 
Englewood,  CO  80110 

1  Psychological  Research  Unit 
Dept,  of  Defence  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

Dr.  Alan  Baddeley 
Medical  Research  Council 
Applied  Research  Council 
15  Chaucer  Road 
Cambridge  CB2  2EF 
England 

Dr.  Patricia  Baggett 
Department  of  Psychology 
University  of  Denver 
University  Park 
Denver,  CO  80208 

Mr.  Avron  Barr 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

Dr.  Isaac  Bejar 
Educational  Testing  Service 
Princeton,  NJ  08450 


Dr.  Warner  Birice 
Streitkaef  teamt 
Rosenberg  5300 
Bonn,  West  Germany  D-5300 

Dr.  R.  Darrel  Bock 
Department  of  Education 
University  of  Chicago 
Chicago,  IL  60637 

Dr.  Nicholas  A.  Bond 
Dept,  of  Psychology 
Sacramento  State  College 
600  Jay  Street 
Sacramento,  CA  95819 

Dr.  Lyle  Bourne 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80302 

Dr.  David  G.  Bowers 
Institute  for  Social  Research 
University  of  .lichigan 
Ann  Arbor,  Ml  48106 

Dr.  Kenneth  Bowles 
Institute  for  Information  Sciences 
University  of  California  at  San  Diego 
La  Jolla,  CA  92037 

Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.0.  Box  168 

Iowa  City,  IA  52240 

Dr.  John  S.  Brown 

XEROX  Palo  Alto  Research  Center 

3333  Coyote  Road 

Palo  Alto,  CA  94304 

Dr.  Bruce  Buchanan 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94304 
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Dr.  C.  Victor  Bunderson 
W i e  a  t  Inc. 

University  Plaza,  Suite  10 
1 160  So.  State  St. 

Orem,  UT  84057 

Dr.  John  B.  Carroll 
Psychometric  Lab 
Univ.  of  No.  Carolina 
Davie  Hal  1  013A 
Chapel  Hill,  NC  27514 

Charles  Meyers  Library 
Livingston  House 
Livingston  Road 
Stratford 
London  El  5  2LJ 
England 

Dr.  Micheline  Chi 
Learning  R&D  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

Dr.  William  Clancey 
Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

Dr.  Kenneth  E.  Clark 
College  of  Arts  and  Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester,  NY  14627 

Dr.  No rma n  Cliff 
Dept,  of  Psychology 
Univ.  of  So.  California 
University  Park 
Los  Angeles,  CA  90007 

Dr.  William  Coffman 
Iowa  Testing  Programs 
University  of  Iowa 
Iowa  City,  IA  52242 


Dr.  Allan  M.  Collins 

Bolt  Beranek  and  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

Dr.  Meredith  Crawford 
Department  of  Engineering 
Administration 

George  Washington  University 
Suite  805 

2101  L  Street  N.  W. 

Washington,  DC  20037 

Dr.  Hans  Cronbag 
Education  Research  Center 
University  of  Leyden 
Boerhaave laan  2 
Leyden 

The  Netherlands 

Major  I.N.  Evonic 
Canadian  Forces  Pers.  Applied 
Research 
1107  Avenue  Road 
Toronto,  Ontario,  Canada 

Dr.  Leonard  Feldt 
Lindquist  Center  for  Measurement 
University  of  Iowa 
Iowa  City,  1A  52242 

Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.0.  Box  168 

Iowa  City,  I A  52240 

Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

Dr.  Gerhardt  Fischer 
Liobigasse  5 
Vienna  1010 
Austria 

Dr.  Donald  Fitzgerald 
University  of  Now  England 
Armidale,  New  South  Wales  2351 
Austral ia 
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Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 

Suite  900 

4330  East  Vest  Highway 
Washington,  DC  20014 

Dr.  John  B.  Frederiksen 
Bolt  Beranek  and  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

Dr.  Robert  Claser 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

Dr.  Ira  Goldstein 

XEROX  Pala  Alto  Research  Center 

3333  Coyote  Road 

Palo  Alto,  CA  94304 

Dr.  Ross  Greene 
Palo  Alto,  CA  94304 
CTB/McGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 

Dr.  James  G.  Greeno 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

Dr.  Alan  Gross 

Center  for  Advanced  Study  in 
Education 

City  University  of  New  York 
New  York,  NY  10036 

Dr.  Ron  Hambleton 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01002 

Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93106 


Dr.  Lloyd  Humphreys 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

Library 

HumRRO/Wes tern  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 

Dr.  Steven  Hunka 
Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
Canada 

Dr.  Earl  Hunt 
Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 

Dr.  Huynh  Huynh 
Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Carl  J.  Jensema 
Gallaudet  College 
Kendall  Green 
Washington,  DC  20002 

Dr.  Arnold  F.  Kanarick 
Honeywell,  Inc. 

2600  Ridgeway  Pkwy 
Minneapolis,  MN  55413 

Dr.  John  A.  Keats 
University  of  Newcastle 
Newcastle,  New  South  Wales 
Australia 

Mr.  Marlin  Kroger 
1117  Via  Goleta 

Palos  Verdes  Estates,  CA  90274 


LCOL.  C.R.J.  Lafleur 
Personnel  Applied  Research 
National  Defence  HQS 
101  Colonel  By  Drive 
Ottawa,  Canada  K1A  0K2 

Dr.  Jill  Larkin 
Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

Dr.  Alan  Lesgold 
Learning  R&D  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  15260 

Dr.  Michael  Levine 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

Faculteit  Sociale  Wetenschappen 
Rijksuniversiteit  Groningen 
Oude  Boteringestraat 
Groningen 
Netherlands 

Dr.  Robert  Linn 
College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Frederick  M.  Lord 
Educational  Testing  Service 
Princeton,  NJ  08540 

Dr.  Robert  R.  Mackie 
Human  Factors  Research,  Inc. 
6780  Cortona  Drive 
Santa  Barbara  Research  Pk. 
Goleta,  CA  93017 

Dr.  Gary  Marco 
Educational  Testing  Service 
Princeton,  NJ  08450 


Dr.  Scott  Maxwell 
Department  of  Psychology 
University  of  Houston 
Houston,  TX  77025 

Dr.  Sam  Mayo 

Loyola  University  of  Chicago 
Chicago,  IL  60601 

Dr.  Mark  Miller 

Systems  and  Information  Sciences  Lab. 
Central  Research  Laboratories 
Texas  Instruments,  Inc. 

Mail  Station  5 
Post  Office  Box  5936 
Dallas,  TX  75222 

Dr.  Richard  B.  Millward 
Dept,  of  Psychology 
Hunter  Lab. 

Brown  University 
Providence,  RI  82912 

Dr.  Allen  Munro 
Univ.  of  So.  California 
Behavioral  Technology  Labs 
3717  South  Hope  Street 
Los  Angeles,  CA  90007 

Dr.  Donald  A.  Norman 
Dept,  of  Psychology  C-009 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

Dr.  Melvin  R.  Novick 
Iowa  Testing  programs 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analysis 
400  Army  Navy  Drive 
Arlington,  VA  22202 

Dr.  Seymour  A.  Papert 
Massachusetts  Institute  of  Technology 
Artificial  Intelligence  Lab. 

545  Technology  Square 
Cambridge,  MA  02139 


Dr.  .tames  A.  Paulson 
Port  1  ami  State  P Diversity 
P.O.  Box  7 *>  1 
Port  land,  OK  0/207 

Mr.  I .ii  i  i  Pet  i  n  1  1  o 
.’•'*11  N.  Kdgowood  Street 
Arlington,  VA  22207 

Dr.  Steven  M.  Pine 
40  SO  Douglas  Avenue 
Col  den  Valiev,  MN  SS41t> 

It  r .  Pot  er  I’o  Ison 
Dept,  ol  Psychology 
Duiversttv  ot  Colorado 
Bon  1  iter,  CO  80302 

Dr.  Diane  M.  Kamsev-Klee 
K-K  Kosoarolt  ami  System  Design 
104/  Kidgomont  Drive 
Malibu,  0A  0026S 

M  In.  Ket  .  M .  Kaueli 
P  It  4 

Btunlosml  n l  st  er  1  urn  Per  Ver t  e i  il  1  gnng 
Pos  t  I  a e h  1  (>  1 
S  t  Bonn  1  ,  Ce rma  nv 

Dr  •  Pot  er  B  •  Rond 
Social  Science  Research  Council 
DOS  Tli  i  ril  Avenue 
New  York,  NY  lOOIh 

Di  .  Mark  I',  Keckase 

Kdiica t  I ona  1  Psychology  Dept. 

Duiversitv  ot  M  ( ssour  i  -Co  luinb  i  a 

14  Hill  Hat  I 

Columbia,  MO  bS2tl| 

|!r.  Kreit  Kell 
SKSAMK 

c/o  Physics  Department 
I'll  l  vers  it  v  ol  Calilornia 
Berkelv,  CA  04/20 


Dr.  Amt  row  M.  Rose 
American  Institutes  lor  Research 
lOSS  Thomas  .letlorson  St.  NW 
Washington,  DC  2000/ 

Dr.  I.eouard  1..  Rosenbaum,  Chairman 
Department  oi  Psycho  logv 
Montgomery  Col  leg1,!' 

Rockville,  MD  208S0 

Dr.  Trust  .  Rothkopt 
Be  1  1  l.ahorat  or  1  es 
BOO  Mountain  Avenue 
Murray  Hill,  N.l  0/0/4 

Dr.  Donald  Rubin 
educational  Testing  Services 
Princeton,  N.l  084 SO 

Dr.  I arrv  Radnor 
C  a  I  1  nude t  Coll ego 
Kenda  1  I  llreen 
Washington,  DC  20002 

Dr.  David  Rome  limi  t 
Center  tor  Human  Inlormat  ion  l’roce; 
Hniv.  ot  Calilornia,  San  Diego 
I. a  Jolla,  CA  0200  1 

Dr.  .1.  Ryan 

Department  ot  Kducat  ion 
University  ot  South  Carolina 
Columbia,  St'  20208 

Prot.  Kumiko  Same.lima 
Dept .  ot  Psychology 
Duiversitv  ol  Tennessee 
Knoxv I  I  I e,  TN  1  70 1 1> 

D i .  A  1 1 en  Schoent e 1 d 
Department  ol  Mat  liemat  les 
11  ami  1 1  on  Co  l  logo 
C  I  1  lit  on,  NY  l  t  12  t 

D r .  Kobe rt  .1.  Seitlel 
I  list  m  l  tonal  Technology  Croup 
llumKKO 

ll'l'  N.  Washington  St. 

Alexandria,  VA  22114 
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Dr.  Kazoa  Shigemasu 

University  of  Tohoku 

Department  of  Educational  Psychology 

Kawauchi,  Sendai  982 

Japan 

Dr.  Edwin  Shirkey 
Department  of  Psychology 
Florida  Technological  University 
Orlando,  FL  32816 

Dr.  Robert  Smith 

Department  of  Computer  Science 

Rutgers  University 

New  Brunswick,  NJ  08903 

Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

Dr.  Albert  Stevens 

Bolt  Beranek  and  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

Dr.  Patrick  Suppes 

Institute  for  Mathmatical  Studies  in 
the  Social  Services 
Stanford  University 
Stanford,  CA  94305 

Dr.  Hariharan  Swaminathan 
Laboratory  of  Psychometeric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Dr.  Brad  Sympson 
Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455 


Dr.  Kikumi  Tatsuoka 
Computer-based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Maurice  Tatsuoka 
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